Introduction to Machine Learning

Unit 08: Wrapper Methods and Feature Extraction with PCA

Introduction

Welcome to Unit 8, where we explore advanced techniques for feature selection and feature extraction.

Today's Focus:

  • Wrapper Methods: Model-based feature selection techniques
  • Principal Component Analysis (PCA): Dimensionality reduction through linear transformation
  • Mathematical Foundations: Eigenvalues, eigenvectors, and covariance matrices

This lecture addresses the curse of dimensionality and provides tools to reduce dimensionality while preserving information.

Theory

Wrapper Methods

Wrapper methods use a predictive model to evaluate feature subsets, using model performance as the selection criterion.

Key Characteristics:

  • Model-based evaluation: Instead of asking "Is this feature statistically relevant?", we ask "Does including this feature improve my model's performance?"
  • Captures feature interactions: Unlike filter methods, wrapper methods can detect interactions between features
  • Algorithm-specific: The selected features are optimal for the specific model being used
  • Trade-off: Higher computational cost for potentially better feature subsets

Common wrapper methods:

Forward Selection

Forward Selection is a greedy algorithm that builds the feature set incrementally:

  1. Start with an empty feature set (Null Model)
  2. Iteratively add features that improve model performance the most
  3. In essence, we fit p simple models (one for each feature) and record each case's accuracy. The feature that produces the best evaluation metric is locked in.
  4. We then add the remaining p-1 features to this feature and select the combination that results in the best evaluation metric at stage 2.
  5. This approach is continued until some stopping rule is satisfied (e.g., desired number of features, no improvement in performance)
Forward Stepwise Selection Example A vertical process diagram showing how variables are progressively added to a statistical model according to significance. Forward Stepwise Selection Example workflow with 5 candidate variables 1 Start with a model with no variables Null Model Baseline with zero predictors Evaluate performance Add the most significant variable Best single feature Model with 1 variable Reassess the model Evaluate performance Add the next most significant variable Best pair of features Model with 2 variables Reassess the model Evaluate performance Continue adding variables... Model with k variables Stopping rule reached Final Model

Characteristics:

  • Greedy: Makes locally optimal choices at each step
  • Irreversible: Once a feature is added, it stays in the model
  • Computationally efficient: Only requires fitting models with increasing feature sets
  • May miss optimal subset: Doesn't consider all possible combinations

Backward Selection

Backward Selection is the reverse of Forward Selection:

  1. Start with all features (Full Model)
  2. Iteratively remove features that hurt performance the least
  3. In essence, we explore different (p - 1) feature combinations and retain the one with the best evaluation score.
  4. From these (p - 1) features, we form pairs of (p - 2) variables, train the model and retain the one with the best evaluation score.
  5. This procedure continues until a stopping rule is reached.
Backward Stepwise Selection Example A vertical process diagram showing backward stepwise selection from a full model with five variables to a final model. Backward Stepwise Selection Example with 5 variables 1 Start with a model containing all variables MODEL Full Model Evaluate performance 2 Remove the least significant variable from the model MODEL Model with 4 variables Evaluate performance 3 Remove the next least significant variable MODEL Model with 3 variables 4 Continue removing variables one at a time MODEL Model with k variables Stopping rule reached Final Model selected based on the stopping rule

Characteristics:

  • Starts with all features: Begins with the complete feature set
  • Removes one at a time: Drops the least important feature at each step
  • Often more computationally intensive: Requires fitting models with decreasing feature sets, often starting with more features
  • May include irrelevant features early: Features that are removed might have been masking the importance of others

Computational Complexity

The computational cost of filter vs. wrapper methods:

Method Complexity Description Example (100 features)
Filter Methods O(d) Evaluate each feature independently 100 evaluations
Wrapper Methods O(d²) Must evaluate feature combinations ~5050 evaluations (50× more expensive)

Note: This analysis focuses on the combinatorial search complexity and doesn't account for the varying computational costs of different filter methods (chi-square vs. mutual information) or ML algorithms (linear regression vs. KNN vs. random forest) used in wrapper methods.

Filter vs Wrapper Comparison

Aspect Filter Wrapper
Speed ✅ Fast ❌ Slow
Model Involvement ❌ No (Algorithm Independent) ✅ Yes (Algorithm Specific)
Captures Feature Interaction ❌ No ✅ Yes
Evaluation Statistical (e.g., Chi-Square, ANOVA) CV Performance
Examples Chi-Square, ANOVA, Mutual Information Forward/Backward Selection, RFE
Overfitting Risk ✅ Low ❌ High

Feature Extraction: Motivation of PCA

The curse of dimensionality presents several challenges:

  • Many ML algorithms struggle with high-dimensional data
  • Example: 100×100 grayscale image = 10,000 features to process
  • Correlated features create redundancy (e.g., height & weight both measure "size")
  • PCA can reduce these to fewer principal components that explain most of the variance
  • High dimensions make visualization impossible. PCA helps reduce to 2D or 3D for easier visualization

Principal Component Analysis (PCA)

PCA is a dimensionality reduction technique that transforms data into a new coordinate system:

  • PCA creates new variables (principal components) that are linear combinations of all original variables
  • Principal components are ordered by how much variance they explain in the data
  • Each principal component is orthogonal (uncorrelated) to all others
  • Effectively removes redundancy from correlated original features

Applications:

Finding Direction of Maximum Variance

PCA aims to find the directions of maximum variance in high-dimensional data and projects the data onto a new subspace with equal or fewer dimensions than the original one.

Key Insight: The orthogonal axes (principal components) of the new subspace can be interpreted as the directions of maximum variance given the constraint that the new feature axes are orthogonal to each other.

Mathematical Note: If vectors \(a\) and \(b\) are orthogonal, then \(a \cdot b = 0\)

Principal component direction of maximum variance A clean infographic showing a two-dimensional elliptical dataset, its first principal component along the major axis, and the preservation of total variance under rotation. Which line points along the direction of maximum variance? Principal Component Analysis finds the axis that best follows the spread of the data. 2D dataset observations Feature 1 Feature 2 First principal component direction of maximum variance 1 Follow the long axis The widest spread is the answer. PCA rotates the coordinate system so that the new axes align with the data's natural directions. original axes PC1 rotation Key property Rotation preserves distances and total variance. TOTAL VARIANCE IS PRESERVED Σ variance of original features = Σ variance of principal components

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are fundamental mathematical concepts used in PCA:

Let \(A\) be a square matrix. A non-zero vector \(x\) is called an eigenvector of \(A\) if and only if there exists a number (real or complex) \(\lambda\) such that: \[ A x = \lambda x \] If such a number \(\lambda\) exists, it is called an eigenvalue of \(A\). The vector \(x\) is called eigenvector associated to the eigenvalue.

Properties:

  • Eigenvectors can only be found for square matrices
  • Not every square matrix has eigenvectors
  • Given an n×n matrix that does have eigenvectors, there are n of them
  • For example, given a \(3 \times 3\) matrix, there are 3 eigenvectors

Mathematical Details: PCA Context

In PCA context, eigenvalues and eigenvectors have special properties:

  • Principal components are eigenvectors of the covariance matrix
  • Eigenvectors from symmetric matrices (like covariance matrices) are automatically orthogonal
  • This guarantees: PC1 · PC2 = 0, PC1 · PC3 = 0, PC2 · PC3 = 0, etc.
  • Practical meaning:
    • Orthogonal = perpendicular = uncorrelated
    • Zero dot product means no linear relationship between components
    • This removes redundancy that existed in original correlated features

PCA Algorithm: Step-by-Step Process

PCA consists of two phases: Learning Phase and Application Phase.

Learning Phase:

  1. Standardize the d-dimensional dataset (mean=0, variance=1 for each feature)
  2. Construct the covariance matrix
  3. Decompose the covariance matrix into its eigenvectors and eigenvalues
  4. Sort the eigenvalues by decreasing order to rank the corresponding eigenvectors

Application Phase:

  1. Select k eigenvectors, which correspond to the k largest eigenvalues, where k is the dimensionality of the new feature subspace (k ≤ d)
  2. Construct a projection matrix \(W\), from the "top" k eigenvectors
  3. Transform the d-dimensional input dataset \(X\), using the projection matrix \(W\), to obtain the new k-dimensional feature subspace

The Transformation Matrix

We construct a \(d \times k\) transformation matrix \(W\) that maps original features to reduced dimensions:

  • Original data point: \(x = [x_1, x_2, ..., x_d] \in \mathbb{R}^d\)
  • Transformation matrix: \(W \in \mathbb{R}^{d \times k}\) (contains k selected eigenvectors as columns)
  • Mathematical operation: \(z = x W\)
  • Transformed data point: \(z = [z_1, z_2, ..., z_k] \in \mathbb{R}^k\)

Example: 13-dimensional wine data → 3-dimensional PCA space

  • W is \(13 \times 3\) matrix (top 3 eigenvectors)
  • Each sample: \([x_1, x_2, \ldots, x_{13}] \rightarrow [\text{PC1\_score}, \text{PC2\_score}, \text{PC3\_score}]\)

Key insight: W contains our selected principal components as columns - this is how eigenvalues/eigenvectors become practical dimensionality reduction!

Key Properties of PCA

  • After transforming original d-dimensional data to k-dimensional subspace (where \(k < d\)), the first principal component captures the maximum possible variance in the data
  • Each subsequent principal component captures the maximum remaining variance while being orthogonal (uncorrelated) to all previous components
  • Even if original features are highly correlated, the resulting principal components are guaranteed to be mutually orthogonal and uncorrelated
  • Critical requirement: Features must be standardized before PCA if they have different scales (e.g., age in years vs. income in dollars), otherwise PCA will be dominated by features with larger numerical ranges

Eigendecomposition: Decomposing a Matrix

Eigendecomposition is at the mathematical core of PCA.

The covariance matrix is a special type of square matrix: it's symmetric, meaning the matrix equals its transpose (\(A = A^T\)).

When we decompose a symmetric matrix, we get valuable properties:

  • Eigenvalues are always real numbers (not complex)
  • Eigenvectors are orthogonal (perpendicular) to each other
  • This guarantees PCA components will be uncorrelated

Key insight: Eigenvalues and eigenvectors come in pairs: each eigenvalue has a corresponding eigenvector that shows the direction of variance.

Most important: The eigenvector with the largest eigenvalue points in the direction of maximum variance in the dataset—this becomes our first principal component.

Constructing the Covariance Matrix

The covariance matrix is a symmetric \(d \times d\) matrix where \(d\) is the number of features in the dataset. It stores pairwise covariances between all features, showing how each pair of features varies together.

Covariance between features \(x_j\) and \(x_k\): \[ \sigma_{jk} = \frac{1}{n - 1}\sum_{i = 1}^{n}(x_j^{(i)} - \mu_j)(x_k^{(i)} - \mu_k) \]

Where \(\mu_j\) and \(\mu_k\) are the sample means of features j and k respectively. Note that the sample means are zero if we standardized the dataset.

  • Positive covariance: Features increase/decrease together
  • Negative covariance: Features vary in opposite directions
  • Zero covariance: Features are uncorrelated

For a 3-feature dataset, the covariance matrix \(\Sigma\) looks like:

\[ \Sigma = \left[ \begin{array}{ccc} \sigma_1^2 & \sigma_{12} & \sigma_{13} \\ \sigma_{21} & \sigma_2^2 & \sigma_{23} \\ \sigma_{31} & \sigma_{32} & \sigma_3^2 \end{array} \right] \]

Important: The diagonal contains variances (\(\sigma_1^2, \sigma_2^2, \sigma_3^2\)), while off-diagonal elements are covariances (\(\sigma_{12}, \sigma_{13}, \sigma_{23}\)).

Explained Variance and Elbow Curve

Each principal component explains a portion of the total variance in the dataset. The explained variance helps determine how many principal components to keep.

Total Variance: The sum of variances of all original features equals the sum of variances of all principal components.

Explained Variance Ratio: The proportion of variance explained by each principal component.

Elbow Curve for Determining Number of Principal Components A scree plot showing explained variance ratios of 45, 30, 15, 7, and 3 percent for five principal components. The elbow occurs around the second or third component. Elbow Curve for Determining Number of Principal Components Identify the point where adding components yields diminishing returns. 1.0 0.75 0.50 0.25 0.0 1 2 3 4 5 Principal Components Explained Variance Ratio MOST INFORMATION RETAINED 45% 30% 15% 7% 3% Elbow point Curve begins to flatten Recommended range Choose 2 or 3 PCs Cumulative information 75–90% Variance by component PC1 45% PC2 30% PC3 15% PC4 7% PC5 3% The elbow balances compression with preservation of signal.

Interactive Examples

PCA Visualization Example

Consider a 2D dataset with correlated features:

Principal Component Analysis Transformation A comparison of highly correlated original data and decorrelated data after PCA transformation. Principal Component Analysis Rotating correlated features into a decorrelated coordinate system Original Data Two highly correlated features x2 x1 Variance is shared across both axes PCA rotate & align After PCA Transformation Data aligned with orthogonal axes PC1 PC2 Components are statistically independent i What PCA preserves PC1 captures the direction of maximum variance; PC2 captures the remaining orthogonal variance.

Forward vs Backward Selection Example

Consider a dataset with 5 features and the following evaluation metrics:

Forward Selection
Backward Selection

Forward Selection Process:

  1. Step 0: Start with empty set, accuracy = 0.60
  2. Step 1: Try each feature individually:
    • F1: accuracy = 0.75 (best)
    • F2: accuracy = 0.70
    • F3: accuracy = 0.72
    • F4: accuracy = 0.65
    • F5: accuracy = 0.68
    → Select F1
  3. Step 2: Try adding each remaining feature to {F1}:
    • {F1, F2}: accuracy = 0.78
    • {F1, F3}: accuracy = 0.82 (best)
    • {F1, F4}: accuracy = 0.76
    • {F1, F5}: accuracy = 0.79
    → Select F3
  4. Step 3: Try adding each remaining feature to {F1, F3}:
    • {F1, F3, F2}: accuracy = 0.85 (best)
    • {F1, F3, F4}: accuracy = 0.80
    • {F1, F3, F5}: accuracy = 0.83
    → Select F2
  5. Step 4: Try adding F4 or F5 to {F1, F3, F2}:
    • {F1, F3, F2, F4}: accuracy = 0.84 (decreases!)
    • {F1, F3, F2, F5}: accuracy = 0.86
    → Select F5

Final selected features: {F1, F3, F2, F5}

Backward Selection Process:

  1. Step 0: Start with all features {F1,F2,F3,F4,F5}, accuracy = 0.86
  2. Step 1: Try removing each feature:
    • Remove F1: accuracy = 0.84
    • Remove F2: accuracy = 0.83
    • Remove F3: accuracy = 0.81 (least impact)
    • Remove F4: accuracy = 0.85
    • Remove F5: accuracy = 0.82
    → Remove F3
  3. Step 2: From {F1,F2,F4,F5}, try removing each:
    • Remove F1: accuracy = 0.82
    • Remove F2: accuracy = 0.80 (least impact)
    • Remove F4: accuracy = 0.83
    • Remove F5: accuracy = 0.81
    → Remove F2
  4. Step 3: From {F1,F4,F5}, try removing each:
    • Remove F1: accuracy = 0.79
    • Remove F4: accuracy = 0.80 (least impact)
    • Remove F5: accuracy = 0.78
    → Remove F4

Final selected features: {F1, F5}

Numerical Solutions

Forward Selection Example

Consider a dataset with 4 features and the following evaluation metrics when adding features one by one:

Step Features Added Accuracy Feature Selected
1{}0.60-
2{F1}0.75F1
3{F1, F3}0.82F3
4{F1, F3, F2}0.85F2
5{F1, F3, F2, F4}0.84F4

Solution:

Selected features: {F1, F3, F2}

Reasoning:

  1. Start with empty set, accuracy = 0.60
  2. Add F1: accuracy improves to 0.75 → F1 selected
  3. Try adding each remaining feature to {F1}:
    • {F1, F2}: accuracy = 0.78
    • {F1, F3}: accuracy = 0.82 (best)
    • {F1, F4}: accuracy = 0.76
    → F3 selected
  4. Try adding each remaining feature to {F1, F3}:
    • {F1, F3, F2}: accuracy = 0.85 (best)
    • {F1, F3, F4}: accuracy = 0.80
    → F2 selected
  5. Try adding F4 to {F1, F3, F2}:
    • {F1, F3, F2, F4}: accuracy = 0.84 (decreases!)
    → Stop here, final features: {F1, F3, F2}

PCA Calculation Example

Given a dataset with the following covariance matrix:

\[ \Sigma = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} \]

Step 1: Find eigenvalues

\[ \det(\Sigma - \lambda I) = (2-\lambda)^2 - 1 = \lambda^2 - 4\lambda + 3 = 0 \] \[ \lambda = \frac{4 \pm \sqrt{16 - 12}}{2} = \frac{4 \pm 2}{2} = 3 \text{ or } 1 \]

Step 2: Find eigenvectors

For \(\lambda_1 = 3\): \[ (\Sigma - 3I)v = \begin{pmatrix} -1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = 0 \] → \(v_1 = v_2\) → Eigenvector: \((1, 1)\) or normalized: \((\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})\)

For \(\lambda_2 = 1\): \[ (\Sigma - I)v = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = 0 \] → \(v_1 = -v_2\) → Eigenvector: \((1, -1)\) or normalized: \((\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})\)

Step 3: Principal Components

PC1: \((\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})\) with eigenvalue 3 (explains more variance)
PC2: \((\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})\) with eigenvalue 1

Try It Yourself

Problem 1: Forward Selection

You have 5 features (F1-F5) and the following accuracy improvements when adding features:

Current FeaturesAdd F1Add F2Add F3Add F4Add F5
{}0.650.700.750.600.68
{F3}0.820.78-0.800.79
{F3, F1}-0.88-0.850.86

Task: Which features would be selected using forward selection with a stopping rule of maximum 3 features?

Solution:

  1. Step 1: Start with empty set. Best single feature is F3 (accuracy 0.75)
  2. Step 2: Add to {F3}. Best improvement is F1 (accuracy 0.82)
  3. Step 3: Add to {F3, F1}. Best improvement is F2 (accuracy 0.88)
  4. Result: Selected features = {F3, F1, F2}
Problem 2: Backward Selection

Using the same dataset as Problem 1, the accuracies when removing features from the full set are:

Current FeaturesRemove F1Remove F2Remove F3Remove F4Remove F5
{F1,F2,F3,F4,F5}0.840.800.750.860.85
{F1,F2,F3,F5}0.830.790.74-0.84
{F1,F2,F3,F5}0.820.780.73--

Task: Which features would be selected using backward selection with a stopping rule of minimum 3 features?

Solution:

  1. Step 1: Start with all features {F1,F2,F3,F4,F5}. Remove F4 (least impact, accuracy 0.86)
  2. Step 2: From {F1,F2,F3,F5}. Remove F5 (accuracy 0.84)
  3. Step 3: From {F1,F2,F3}. Stop (reached minimum of 3 features)
  4. Result: Selected features = {F1, F2, F3}
Problem 3: PCA Eigenvalue Calculation

Given the covariance matrix:

\[ \Sigma = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix} \]

Tasks:

  1. Find the eigenvalues
  2. Which principal component explains more variance?
  3. What proportion of total variance does each PC explain?

Solution:

  1. Eigenvalues:
    \[ \det(\Sigma - \lambda I) = (3-\lambda)^2 - 1 = \lambda^2 - 6\lambda + 8 = 0 \] \[ \lambda = \frac{6 \pm \sqrt{36 - 32}}{2} = \frac{6 \pm 2}{2} = 4 \text{ or } 2 \]
  2. Variance explanation: PC1 (λ=4) explains more variance than PC2 (λ=2)
  3. Proportions:
    • PC1: 4/(4+2) = 4/6 = 66.7%
    • PC2: 2/6 = 33.3%
Problem 4: Standardization Importance

You have a dataset with two features:

  • Age: values range from 18 to 80 (mean=45, std=15)
  • Income: values range from $20,000 to $200,000 (mean=$80,000, std=$40,000)

Task: What happens if you apply PCA without standardizing the data first? Which feature will dominate the first principal component?

Solution:

Problem: Without standardization, PCA will be dominated by the feature with the largest scale (Income).

Why: PCA maximizes variance. Income has much larger absolute values and thus larger variance in the original scale, even though the relative variability (coefficient of variation) might be similar.

Result: The first principal component will be heavily weighted toward Income, and Age will have minimal influence.

Solution: Always standardize features (mean=0, variance=1) before applying PCA when features have different scales.

Problem 5: Explained Variance

After applying PCA to a dataset with 10 features, you get the following eigenvalues for the principal components:

[4.5, 3.2, 1.8, 0.9, 0.6, 0.3, 0.2, 0.1, 0.05, 0.05]

Tasks:

  1. What is the total variance in the dataset?
  2. How much variance is explained by the first 3 principal components?
  3. What proportion of total variance is explained by the first 3 PCs?
  4. How many PCs would you keep to explain at least 90% of the variance?

Solution:

  1. Total variance: Sum of all eigenvalues = 4.5 + 3.2 + 1.8 + 0.9 + 0.6 + 0.3 + 0.2 + 0.1 + 0.05 + 0.05 = 11.7
  2. Variance by first 3 PCs: 4.5 + 3.2 + 1.8 = 9.5
  3. Proportion: 9.5 / 11.7 ≈ 0.812 or 81.2%
  4. PCs for 90%:
    • PC1: 4.5/11.7 ≈ 38.5%
    • PC1+PC2: (4.5+3.2)/11.7 ≈ 65.8%
    • PC1+PC2+PC3: 81.2%
    • PC1+PC2+PC3+PC4: (9.5+0.9)/11.7 ≈ 89.7%
    • PC1+PC2+PC3+PC4+PC5: (10.1+0.6)/11.7 ≈ 91.5%
    → Need 5 PCs to explain ≥90% of variance

Interactive Quiz

Test your understanding of Wrapper Methods and PCA:

Question 1: What is the main advantage of wrapper methods over filter methods?

A) They are faster to compute
B) They capture feature interactions
C) They are algorithm-independent
D) They use statistical tests

Question 2: In forward selection, once a feature is added to the model, can it be removed later?

A) Yes, if it becomes less important
B) No, forward selection only adds features
C) Yes, but only in the final step
D) It depends on the stopping criterion

Question 3: What is the primary purpose of PCA?

A) To increase the number of features
B) To reduce dimensionality while preserving variance
C) To make features correlated
D) To select the most important features

Question 4: Which of the following is NOT a property of principal components?

A) They are orthogonal to each other
B) They are linear combinations of the original features
C) They are ordered by the amount of variance they explain
D) They are correlated with each other

Question 5: Why is it important to standardize features before applying PCA?

A) To make the mean of each feature zero
B) To ensure features with larger scales don't dominate the principal components
C) To make the covariance matrix diagonal
D) To increase computational efficiency

Key Takeaways

Wrapper Methods:

  • Model-based evaluation: Uses a predictive model to evaluate feature subsets based on performance
  • Captures interactions: Can detect interactions between features that filter methods might miss
  • Algorithm-specific: Selected features are optimal for the specific model being used
  • Computationally expensive: Requires training multiple models, leading to O(d²) complexity

Forward vs Backward Selection:

  • Forward Selection: Starts with empty set, adds features one by one. More efficient for large feature sets.
  • Backward Selection: Starts with all features, removes least important one by one. Often more computationally intensive.
  • Both are greedy: Make locally optimal choices at each step, may not find global optimum

Principal Component Analysis (PCA):

  • Dimensionality reduction: Transforms data into lower-dimensional space while preserving variance
  • Linear transformation: Creates new variables (principal components) as linear combinations of original features
  • Orthogonal components: Principal components are uncorrelated (orthogonal) to each other
  • Variance maximization: First PC captures maximum variance, each subsequent PC captures maximum remaining variance
  • Eigen decomposition: PCs are eigenvectors of the covariance matrix, ordered by their eigenvalues

PCA Key Properties:

  • Standardization required: Features must be standardized (mean=0, variance=1) if they have different scales
  • Variance preservation: Total variance is preserved: sum of original variances = sum of PC variances
  • Information retention: First few PCs often capture most of the information in the data
  • Decorrelation: Transforms correlated features into uncorrelated principal components

Common Pitfalls

⚠️ Wrapper Methods:

  • Computational cost: Wrapper methods can be very slow for datasets with many features due to O(d²) complexity
  • Overfitting: Using the same data for both feature selection and model training can lead to overfitting. Always use cross-validation.
  • Model dependency: Selected features are optimal for the specific model used, may not generalize to other models
  • Greedy nature: Both forward and backward selection make locally optimal choices, which may not lead to the global optimum
  • Stopping criteria: Choosing the wrong stopping rule can lead to underfitting (too few features) or overfitting (too many features)

⚠️ PCA:

  • Interpretability loss: Principal components are linear combinations of original features, making them less interpretable
  • Non-linear relationships: PCA is a linear transformation and may not capture non-linear relationships in the data
  • Standardization requirement: Forgetting to standardize features with different scales will cause PCA to be dominated by high-variance features
  • Information loss: Reducing dimensionality always loses some information. Need to choose k carefully to balance reduction vs. information retention
  • Outlier sensitivity: PCA is sensitive to outliers, which can disproportionately influence the principal components
  • Zero variance features: Features with zero variance will cause numerical instability in the covariance matrix
  • Choosing k: No universal rule for choosing the number of components. The elbow method is subjective.

⚠️ General:

  • Feature selection vs extraction: Don't confuse the two. Selection keeps original features, extraction creates new ones.
  • Data leakage: Applying feature selection/extraction before train-test split can leak information from test to train
  • Correlation assumption: PCA works best when features are correlated. If features are already uncorrelated, PCA may not provide much benefit.

Resources

📚 Wrapper Methods:

📚 PCA:

📚 Mathematical Foundations:

📖 Books:

💻 Practical Implementation: